Now that we know how to model urges using heart rate, we can compute models for each participant and for the global dataset.
Here, we order the values prioritising accuracy, followed by sensitivity (ability to correctly detect an urge/episode), and lastly specificity (ability to correctly reject non-urges).
From this, I wonder how we can decide the best model…
We can see that random forest is quite popular (even though its global prediction was low, as compared to this subset):
According to this plot, the random forest consistently performs best (judging by the mean). It is only outperformed in sensitivity, by the neural net (which has poorest mean accuracy and mean specificity) and the Naive Bayesian Tree (which also has poor mean specificity by comparison).
Random forest was able to achieve 100% prediction results on participant 200, and performed poorly with participant 205.
First, we compare the model specifications:
By these plots, we can agree that the variables sdnn, sdann, and sdnnix are probably most important as they are found highest both in the global and best model. Let’s confirm by ranking the variable importance in each model:
[1] "Ranks for variable `hr`:"
----------
Rank n
------ ---
7 4
5 3
3 2
6 2
8 2
9 2
1 1
2 1
4 1
----------
[1] "Ranks for variable `meanrr`:"
----------
Rank n
------ ---
2 4
5 4
6 3
4 2
7 2
8 2
1 1
----------
[1] "Ranks for variable `sdnn`:"
----------
Rank n
------ ---
2 8
3 4
6 2
1 1
4 1
8 1
10 1
----------
[1] "Ranks for variable `sdann`:"
-----------
Rank n
------ ----
1 12
2 2
3 2
4 2
-----------
[1] "Ranks for variable `sdnnix`:"
----------
Rank n
------ ---
5 4
1 3
3 3
4 2
9 2
2 1
6 1
7 1
8 1
----------
[1] "Ranks for variable `pnn50`:"
----------
Rank n
------ ---
4 4
6 4
3 3
5 3
2 1
7 1
9 1
10 1
----------
[1] "Ranks for variable `sdsd`:"
----------
Rank n
------ ---
9 5
8 4
5 2
6 2
7 2
3 1
4 1
10 1
----------
[1] "Ranks for variable `rmssd`:"
----------
Rank n
------ ---
8 6
9 4
3 2
4 2
7 2
6 1
10 1
----------
[1] "Ranks for variable `irrr`:"
----------
Rank n
------ ---
7 5
9 4
4 3
6 2
8 2
5 1
10 1
----------
[1] "Ranks for variable `madrr`:"
-----------
Rank n
------ ----
10 13
2 1
3 1
5 1
6 1
7 1
-----------
We can assume that participant 205 had the poorest predictions because their model selected poor variables. What do these variables look like?
Here’s what the global data set looks like:
[1] 2199 11
hr meanrr sdnn sdann sdnnix pnn50
Min. : 45.13 Min. : 372.4 Min. : 24.29 Min. : 0.00 Min. : 2.293 Min. :0.0000
1st Qu.: 72.54 1st Qu.: 696.0 1st Qu.: 53.57 1st Qu.: 15.22 1st Qu.: 46.073 1st Qu.:0.1088
Median : 79.33 Median : 762.8 Median : 64.39 Median : 26.76 Median : 56.843 Median :0.1606
Mean : 81.03 Mean : 766.6 Mean : 69.86 Mean : 34.90 Mean : 59.880 Mean :0.1673
3rd Qu.: 87.04 3rd Qu.: 833.5 3rd Qu.: 80.17 3rd Qu.: 45.98 3rd Qu.: 69.031 3rd Qu.:0.2258
Max. :162.51 Max. :1332.5 Max. :452.62 Max. :263.32 Max. :586.990 Max. :0.6000
sdsd rmssd irrr madrr Episodes
Min. : 18.71 Min. : 18.70 Min. : 11.72 Min. :-39.0647 0:1037
1st Qu.: 53.95 1st Qu.: 53.86 1st Qu.: 62.50 1st Qu.: 0.0000 1:1162
Median : 66.51 Median : 66.21 Median : 62.50 Median : 0.0000
Mean : 71.63 Mean : 71.23 Mean : 77.04 Mean : 1.0660
3rd Qu.: 82.94 3rd Qu.: 82.54 3rd Qu.: 93.75 3rd Qu.: 0.0006
Max. :502.60 Max. :491.55 Max. :312.51 Max. : 93.7538
Here’s participant 200:
[1] 21 11
hr meanrr sdnn sdann sdnnix pnn50
Min. : 65.34 Min. :608.8 Min. : 47.01 Min. : 2.141 Min. : 40.13 Min. :0.07312
1st Qu.: 79.25 1st Qu.:685.4 1st Qu.: 60.43 1st Qu.: 20.557 1st Qu.: 56.24 1st Qu.:0.16667
Median : 81.15 Median :750.9 Median : 80.69 Median : 32.198 Median : 63.72 Median :0.20183
Mean : 83.08 Mean :745.1 Mean : 86.25 Mean : 49.430 Mean : 75.14 Mean :0.20711
3rd Qu.: 89.23 3rd Qu.:760.6 3rd Qu.: 87.92 3rd Qu.: 45.186 3rd Qu.: 77.41 3rd Qu.:0.27193
Max. :100.53 Max. :939.8 Max. :209.57 Max. :185.514 Max. :183.63 Max. :0.33333
sdsd rmssd irrr madrr Episodes
Min. : 42.43 Min. : 42.42 Min. : 62.50 Min. :-23.438573 0: 8
1st Qu.: 63.03 1st Qu.: 62.96 1st Qu.: 74.22 1st Qu.: 0.000000 1:13
Median : 78.50 Median : 78.31 Median : 93.75 Median : 0.000000
Mean : 80.12 Mean : 79.78 Mean : 95.61 Mean : 1.116191
3rd Qu.: 94.27 3rd Qu.: 93.65 3rd Qu.:125.01 3rd Qu.: 0.000954
Max. :144.77 Max. :143.62 Max. :156.26 Max. : 15.625834
And participant 205:
[1] 15 11
hr meanrr sdnn sdann sdnnix pnn50
Min. : 75.37 Min. :427.6 Min. : 53.46 Min. : 8.355 Min. :33.23 Min. :0.09091
1st Qu.: 83.17 1st Qu.:663.9 1st Qu.: 76.75 1st Qu.:28.537 1st Qu.:64.08 1st Qu.:0.20000
Median : 85.67 Median :707.9 Median : 81.81 Median :37.346 Median :69.13 Median :0.24194
Mean : 92.91 Mean :677.3 Mean : 86.12 Mean :45.911 Mean :69.45 Mean :0.22682
3rd Qu.: 91.22 3rd Qu.:733.3 3rd Qu.: 95.51 3rd Qu.:49.529 3rd Qu.:78.48 3rd Qu.:0.26491
Max. :142.86 Max. :813.2 Max. :118.79 Max. :99.376 Max. :96.78 Max. :0.28671
sdsd rmssd irrr madrr Episodes
Min. : 57.15 Min. : 54.58 Min. : 39.06 Min. :-31.250715 0:7
1st Qu.: 69.55 1st Qu.: 69.29 1st Qu.: 78.13 1st Qu.: -7.812500 1:8
Median : 84.69 Median : 84.01 Median : 93.75 Median : 0.000238
Mean : 85.09 Mean : 84.38 Mean : 93.75 Mean : -2.083309
3rd Qu.: 91.83 3rd Qu.: 91.28 3rd Qu.:115.24 3rd Qu.: 3.907025
Max. :133.22 Max. :131.84 Max. :156.26 Max. : 15.625000
We can examine the distributions of the three top importance variables too:
Notice the difference in the ranges of these variables.
In any case, both participants 200 and 205 have extremely small data sets; changing any parameters such as the random seed could easily invalidate these results. Let’s examine the second best predictions, participant 207 (who we examined before).
Once again, our top three variables are present.
I’m noticing that the good models all lack heavy tails. Participant 207’s data is very similar to 200, however, it has a tiny bump way out at 400 for sdnn; by comparison, participant 200 had a range within 250 for these three variables. Does this mean that our preprocessing should apply a ceiling to these three variables?
Random forest is a simple algorithm with a complex explanation; it’s considered one of the more black box algorithms, as it’s not common to get too much detail about mechanics out of it. Nevertheless, we can derive some information by randomly selecting a tree and checking it’s mechanics. This is not how the overall model works though (the real algorithm draws samples repeatedly and builds on the errors of small decision trees).